jpeg2000: valid_file implementation, much faster than trying to open #4548
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Primary cost of detecting whether a given file is valid jpeg2000 file is the thread pool initialization and shutdown that Jpeg2000Input::open does (the actual thread pool part is inside OpenJpeg code). So add a dedicated valid_file() implementation that only needs to check 12 bytes of the header. While at it, I changed already existing isJp2File() to is_jp2_header() to better match naming conventions used elsewhere, and instead of trying to handle both little and big endian cases by manual repetition of two sets of magic integers, let's do just byte comparisons with memcmp instead. On my PC (Ryzen 5950X, SSD, Windows VS2022), doing ImageInput::create() on 1138 files where they are not images at all (so OIIO in turns tries all the input plugins on them): - Before: 3.4 seconds spent in Jpeg2000Input::open (1.9s opj_thread_pool_create, 1.3s opj_thread_pool_destroy) - After: 33 milliseconds spent in Jpeg2000Input::valid_file Signed-off-by: Aras Pranckevicius <aras@nesnausk.org>
lgritz
approved these changes
Dec 1, 2024
|
Oof, the OpenJPEG library creates and destroys a threadpool for every file it reads (or tries to read)? It really doesn't just let the thread pool persist so that if multiple files are read, it doesn't do a full build/teardown each time? |
|
Always good to see you helping out here, @aras-p. You never fail to turn up some really interesting thing that we had previously overlooked. |
Looks like it. It's... not a design I would recommend, let's put it that way :) |
lgritz
pushed a commit
to lgritz/OpenImageIO
that referenced
this pull request
Dec 1, 2024
…open (AcademySoftwareFoundation#4548) Primary cost of detecting whether a given file is valid jpeg2000 file is the thread pool initialization and shutdown that `Jpeg2000Input::open` does (the actual thread pool part is inside OpenJpeg code). So add a dedicated `valid_file()` implementation that only needs to check 12 bytes of the header. While at it, I changed already existing `isJp2File()` to `is_jp2_header()` to better match naming conventions used elsewhere, and instead of trying to handle both little and big endian cases by manual repetition of two sets of magic integers, let's do just byte comparisons with `memcmp` instead. On my PC (Ryzen 5950X, SSD, Windows VS2022), doing `ImageInput::create()` on 1138 files where they are not images at all (so OIIO in turns tries all the input plugins on them): - Before: **3.4 seconds** spent in `Jpeg2000Input::open` (1.9s `opj_thread_pool_create`, 1.3s `opj_thread_pool_destroy`) - After: **33 milliseconds** spent in `Jpeg2000Input::valid_file` No new tests. I checked behavior on the official Jpeg2000 conformance data set (https://github.com/uclouvain/openjpeg-data/tree/master/input/conformance), seems to work. Signed-off-by: Aras Pranckevicius <aras@nesnausk.org>
lgritz
pushed a commit
to lgritz/OpenImageIO
that referenced
this pull request
Dec 9, 2024
…open (AcademySoftwareFoundation#4548) Primary cost of detecting whether a given file is valid jpeg2000 file is the thread pool initialization and shutdown that `Jpeg2000Input::open` does (the actual thread pool part is inside OpenJpeg code). So add a dedicated `valid_file()` implementation that only needs to check 12 bytes of the header. While at it, I changed already existing `isJp2File()` to `is_jp2_header()` to better match naming conventions used elsewhere, and instead of trying to handle both little and big endian cases by manual repetition of two sets of magic integers, let's do just byte comparisons with `memcmp` instead. On my PC (Ryzen 5950X, SSD, Windows VS2022), doing `ImageInput::create()` on 1138 files where they are not images at all (so OIIO in turns tries all the input plugins on them): - Before: **3.4 seconds** spent in `Jpeg2000Input::open` (1.9s `opj_thread_pool_create`, 1.3s `opj_thread_pool_destroy`) - After: **33 milliseconds** spent in `Jpeg2000Input::valid_file` No new tests. I checked behavior on the official Jpeg2000 conformance data set (https://github.com/uclouvain/openjpeg-data/tree/master/input/conformance), seems to work. Signed-off-by: Aras Pranckevicius <aras@nesnausk.org>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Primary cost of detecting whether a given file is valid jpeg2000 file is the thread pool initialization and shutdown that
Jpeg2000Input::opendoes (the actual thread pool part is inside OpenJpeg code).So add a dedicated
valid_file()implementation that only needs to check 12 bytes of the header. While at it, I changed already existingisJp2File()tois_jp2_header()to better match naming conventions used elsewhere, and instead of trying to handle both little and big endian cases by manual repetition of two sets of magic integers, let's do just byte comparisons withmemcmpinstead.On my PC (Ryzen 5950X, SSD, Windows VS2022), doing
ImageInput::create()on 1138 files where they are not images at all (so OIIO in turns tries all the input plugins on them):Jpeg2000Input::open(1.9sopj_thread_pool_create, 1.3sopj_thread_pool_destroy)Jpeg2000Input::valid_fileTests
No new tests. I checked behavior on the official Jpeg2000 conformance data set (https://github.com/uclouvain/openjpeg-data/tree/master/input/conformance), seems to work.
Checklist:
need to update the documentation, for example if this is a bug fix that
doesn't change the API.)
(adding new test cases if necessary).
corresponding Python bindings (and if altering ImageBufAlgo functions, also
exposed the new functionality as oiiotool options).
already run clang-format before submitting, I definitely will look at the CI
test that runs clang-format and fix anything that it highlights as being
nonconforming.